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Executive Summary 

This report summarizes an evaluation study investigating the effects of participation 
in the 21 st Century Community Learning Centers (CCLC) program on student achievement in 
mathematics and reading/language arts, for the cohort of students who participated during 
the 2012-2013 school year. The report is a supplement to the Office of Assessment, Ac- 
countability, and Research’s annual evaluation of the CCLC program. 

Methods 

We conducted a quasi-experimental examination of within- and between-group dif- 
ferences in student assessment outcomes in both mathematics and reading/language arts. 
The study addressed 10 research questions (RQs). The treatment group consisted of students 
who participated in at least 30 days of CCLC during the 2012-2013 school year. A control 
group consisting of demographically similar students who did not participate in CCLC was 
selected using propensity score matching (PSM). 

Findings 

Research Questions 1 and 2 (RQ1 and RQ2) 

RQi and RQ2 sought to determine if CCLC and non-CCLC students’ year-to-year 
gains in mathematics and reading/language arts were significantly different. These ques- 
tions were addressed using a series of independent samples t-tests. The analyses used group 
membership as the independent variable and mean WESTEST 2 scale score gains from 

2011- 2012 to 2012-2013 in mathematics and reading/language arts as the outcome varia- 
bles. Results showed no significant differences between these groups in either mathematics 
or reading/language arts scale score gains. 

Research Questions 3 and 4 (RQ3 and RQ4) 

RQ3 and RQ4 sought to determine if CCLC and non-CCLC students’ 2012-2013 end- 
of-year scores in mathematics and reading/language arts were significantly different. These 
questions were addressed using a series of independent samples t-tests. The analyses used 
group membership as the independent variable and mean WESTEST 2 scale score from the 

2012- 2013 school year in mathematics and reading/language arts as the outcome variables. 
Results showed no significant differences between these groups in either mathematics or 
reading/language arts scale scores. 
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Research Questions 5 and 6 (RQ5 and RQ6) 

RQ5 and RQ6 sought to determine if CCLC students experienced statistically signifi- 
cant changes in performance from 2011-2012 to 2012-2013. These questions were ad- 
dressed using a series of paired t-tests. The analyses used time as the independent variable 
and CCLC students’ mean 2011-2012 and 2012-2013 WESTEST 2 scale scores in mathe- 
matics and reading/language arts as the outcome variables. CCLC students exhibited statis- 
tically significant mathematics gains in Grades 4, 5, 6, 7, 9 and 11; the students also 
exhibited statistically significant gains in reading/language arts in Grades 4, 5, 6, 7, 8, and 
10. 

Research Questions 7 and 8 (RQ7 and RQ8) 

RQ7 and RQ8 sought to determine if non-CCLC students experienced statistically 
significant changes in performance from 2011-2012 to 2012-2013. These questions were 
addressed using a series of paired t-tests. The analyses used time as the independent varia- 
ble and non-CCLC students’ mean 2011-2012 and 2012-2013 WESTEST 2 scale scores in 
mathematics and reading/language arts as the outcome variables. Similar to CCLC students, 
non-CCLC students exhibited statistically significant mathematics gains in several grades 
(i.e., Grades 4, 5, 6, 7, 8, 9 and 11), and reading/language arts gains in Grades 4, 5, 6, and 7. 

Research Questions 9 and 10 (RQ9 and RQ10) 

RQ9 and RQ10 sought to determine if student achievement over time differed signifi- 
cantly between groups. These questions were addressed using repeated measures analysis of 
variance (RM ANOVA) tests. The analyses used two predictor variables, group membership 
and time, as independent variables predicting the outcome of WESTEST 2 performance in 
mathematics and reading language arts. We looked for a significant interaction effect to in- 
dicate one group scored differently from the other over time. We found significant main ef- 
fects for time, but there were no statistically significant interaction effects between groups 
and time. However, in reading/language arts, the interaction effects approached significance 
in Grades 8 and 10. In both cases, CCLC students outperformed the non-CCLC comparison 
group. 

Conclusions 

There were no statistically significant differences between groups. When examining 
within-group differences, both groups exhibited multiple statistically significant changes in 
mathematics and reading/language arts performance. However, only in the case of Grades 8 
and 10 reading/language arts did the results approach statistical significance in the predict- 
ed direction (i.e., with CCLC students outperforming non-CCLC students). 

Limitations of study 

This study had several important limitations that may limit our ability to draw defini- 
tive conclusions about the effectiveness of the CCLC program in producing academic 
achievement gains. First, this study only encompassed a single year of CCLC intervention. It 
is likely that academic achievement gains on standardized assessments would not be realized 
until more time has elapsed. Second, we were able to examine only Grades 4-11 in the study 
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due to a lack of available achievement data for Grades K-3. This is a significant limitation 
when one considers the fact that approximately 50% of the 2012-2013 CCLC cohort was en- 
rolled in these grades. 

Recommendations 

To the extent possible, we will attempt to prepare next year’s edition of this report at 
the outset of the 2014-2015 school year, a time when the data are more actionable for CCLC 
program staff. Further, we will work with CCLC program staff to plan additional research to 
be conducted at the conclusion of the 2013-2014 school year to examine the impact of long- 
er-term participation in CCLC on student academic achievement outcomes. The study will 
examine outcomes for students who participated in CCLC for at least 2 academic years to 
determine if prolonged participation in the program produces statistically significant gains 
in achievement. 
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Introduction 


This year the West Virginia Department of Education Office of Assessment, Account- 
ability and Research is providing two separate evaluation reports for the 21 st Century Com- 
munity Learning Centers Program (CCLC). The first, a descriptive evaluation of the 
program’s implementation and key performance measures, we have provided in the past. 
This report, the second, employs a quasi-experimental design to evaluate the impact of 21 st 
CCLC program participation on student academic achievement. The new model is being 
used to extract evidence of the effect of the CCLC program on reliable quantitative academic 
assessments of participating students that were in the program in 2013. 

The explanatory variable for this study was defined as participation in the CCLC pro- 
gram for at least 30 days. Students meeting this criterion were identified by teachers who 
completed the 2013 CCLC teacher survey. The outcome variables used in this study are de- 
rived from the West Virginia Educational Standards Test 2 (WESTEST 2). WESTEST 2 is a 
summative test consisting of four content areas (mathematics, reading/language arts, sci- 
ence, and social studies); all students in Grades 3-11 take the WESTEST 2. The West Virgin- 
ia Department of Education administers WESTEST 2 annually. This study examined 
academic performance at the conclusion of the 2012-2013 school year and academic 
achievement gains realized from 2011-2012 to 2012-2013 in both mathematics and read- 
ing/language arts for Grades 4-11 1 . The treatment group includes students participating in 
the CCLC program and the control group includes nonparticipating matched students. 

To examine the impact of CCLC program participation on 2-year academic gains, the 
following research questions were posed. 

RQi Do students participating in the 2013 CCLC exhibit significantly higher mean 
scale score gains in WESTEST 2 mathematics in each grade, 4-11, than non- 
participating students in each of these grades from 2012 to 2013? 

RQ2 Do students participating in the 2013 CCLC exhibit significantly higher mean 
scale score gains in WESTEST 2 reading/language arts in each grade, 4-11, 
than nonparticipating students in each of these grades from 2012 to 2013? 

To examine the impact of CCLC program participation on end-of-year academic 
achievement, the following research questions were posed. 

RQ3 Do students participating in the 2013 CCLC exhibit significantly higher mean 
scale scores in WESTEST 2 mathematics in 2013 in each grade, 4-11, than 
nonparticipating students in each of these grades? 


1 Grades K-2 are excluded due to the lack of a common outcome metric for these students. 
Grade 3 is excluded because the evaluation design required controlling for prior academic achieve- 
ment. 
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RQ4 Do students participating in the 2013 CCLC exhibit significantly higher mean 
scale scores in WESTEST 2 reading/language arts in 2013 in each grade, 4- 
11, than nonparticipating students in these grades? 

We also examined the year-to-year changes in academic achievement for each group 
independently. The following research questions were posed. 

RQ5 Do students participating in CCLC exhibit significant change in mean scale 
scores in WESTEST 2 mathematics by grade over time? 

RQ6 Do students participating in CCLC exhibit significant change in mean scale 
scores in WESTEST 2 reading/language arts by grade over time? 

RQ7 Do students not participating in CCLC exhibit significant change in mean 
scale scores in WESTEST 2 mathematics by grade over time? 

RQ8 Do students not participating in CCLC exhibit significant change in mean 
scale scores in WESTEST 2 reading/language arts by grade over time? 

Finally, we sought to determine if gains experienced by CCLC students were signifi- 
cantly different from those gains experienced by non-CCLC students. The following research 
questions were posed. 

RQ9 Do students participating in CCLC exhibit significantly higher mean scale 
scores in WESTEST 2 mathematics than the nonparticipating students by 
grade over time? 

RQ10 Do students participating in CCLC exhibit significantly higher mean scale 
scores in WESTEST 2 reading/language arts than the nonparticipating stu- 
dents by grade over time? 


Methods 

We conducted a quasi-experimental examination of existing student assessment data 
in mathematics and reading/language arts for students who participated in CCLC during the 
2012-2013 school year. The treatment group consisted of students who participated in at 
least 30 days of CCLC intervention as reported by educators. The comparison group was 
matched using propensity score matching using a variety of demographic and performance 
covariates. Analyses were conducted to examine both within- and between-group differences 
in student achievement. 


Population Characteristics 

The West Virginia Department of Education provided CCLC teachers an online sur- 
vey to be completed about each student who participated in the program during the 2012- 
2013 school year. Among other items, the teacher survey asked educators to identify stu- 
dents who met minimum criteria for CCLC program participation. To be selected for inclu- 
sion in the treatment group for this study, students had to be enrolled in an assessed grade 
level during this school year and have participated in the CCLC program for 30 days or 
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more. A matched comparison group was selected from the population of all non-CCLC stu- 
dents. 


Sampling procedures 

From the teacher survey, we identified more than 4,000 students who participated in 
the CCLC program during the 2012-2013 school year. The criteria for inclusion in this study 
included having (a) a valid unique student identifier, (b) grade level assessment scale scores 
for the last 2 years in both mathematics and reading/language arts, and (c) having a com- 
plete set of demographic covariate variables to be used during matching. After removing 
students in Grades K-3 and those for whom we did not receive a valid unique student identi- 
fier, we were left with a pool of 2,246 potential students. Only 1,671 of these students had 
test records for the 2 years needed (74%). After removing the remainder of students who did 
not receive scale scores for both mathematics and reading/language arts and who were re- 
tained from one year to the next, we were left with a final sample of 1,648 CCLC students. 
This represents 73% of the initial pool of valid student records. 

We used propensity score matching (PSM) to select a matched comparison group for 
each grade level. This methodology uses logistic regression to select a comparison group that 
closely matches the treatment group on a variety of observed covariates. First, a binary indi- 
cator was created to indicate whether or not each student in the state participated in CCLC 
during the 2012-2013 school year. Group 1 was defined as the treatment group (those stu- 
dents who attended 30 or more days in the CCLC program) and Group o was defined as the 
control group (those students who did not participate in the CCLC program during the 
2012-2013 school year). We then specified the PSM models, which derived conditional 
probabilities for each student by regressing the binary group membership variable on the 
following covariates: (a) prior academic achievement in both mathematics and reading/ 
language arts, (b) gender, (c) race/ ethnicity, (d) free/reduced-price lunch eligibility, and (e) 
special education eligibility. Grade level was held constant by conducting matching inde- 
pendently within each grade. Thus, in this study the propensity score represented the pre- 
dicted probability that a given student would attend 30 days of CCLC based on this set of 
pre-intervention covariates. Finally, we used nearest neighbor matching to select the most 
appropriate match for each CCLC student. Verification analyses revealed that this matching 
methodology identified an adequately balanced comparison group for hypothesis testing 2 . 

Measures and Covariates 

This study includes an examination of student achievement data. We analyzed indi- 
vidual students’ scale scores, and gain scores in both mathematics and reading language 
arts. Gain scores were operationalized as the change in student scale scores from 2011-2012 
to 2012-2013. This time period was selected because the 2012-2013 school year represented 
the implementation year for the CCLC program. Thus, it was reasonable to expect that stu- 


2 We used chi squared analyses to verify the two groups did not differ on categorical demo- 
graphic variables. We used independent samples t-tests to verify the two groups did not differ on pri- 
or academic achievement in reading/language arts and mathematics. 
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dents who participated in CCLC during the 2012-2013 school year would experience differ- 
ential gains when compared with similar students not known to have participated in CCLC. 

Covariates utilized in this study include students’ gender, race/ ethnicity, special edu- 
cation eligibility, free/reduced-price lunch eligibility, and prior academic achievement in 
mathematics and reading/language arts. 

Data collection methods 

All data for this study were collected from two sources— the CCLC Teacher Survey 
and the West Virginia Educational Information System (WVEIS). CCLC Teacher Survey data 
were collected by the researchers as part of the annual descriptive evaluation of the 21 st 
CCLC Program. WVEIS data were extracted from the annual testing record file collected by 
the West Virginia Department of Education. 

Research Design 

RQi and RQ2 were addressed using a series of independent samples t tests. These 
analyses used group membership as the independent variable and mean WESTEST 2 scale 
score gains from 2011-2012 to 2012-2013 in mathematics and reading/language arts as the 
outcome variables. Each grade level and content area combination was tested independently 
to estimate impact of the CCLC program. In sum, we conducted 16 tests: 

1. Eight tests (one per grade for Grades 4-11) examined the impact of group member- 
ship on year-to-year WESTEST 2 mathematics gains to determine whether CCLC 
students (treatment) experienced greater gains than students who did not receive the 
treatment (control). 

2. Eight tests (one per grade for Grades 4-11) examined the impact of group member- 
ship on year-to-year WESTEST 2 reading/language arts gains to determine whether 
CCLC students (treatment) experienced greater gains than students who did not re- 
ceive the treatment (control). 

RQ3 and RQ4 were addressed using a series of independent samples t tests. These 
analyses used group membership as the independent variable and mean 2012-2013 
WESTEST 2 outcomes in mathematics and reading/language arts as the outcome variables. 
Each grade level and content area combination was tested independently to estimate impact 
of the CCLC program in 2013. In sum, we conducted 16 tests: 

3. Eight tests (one per grade for Grades 4-11) examined the impact of group member- 
ship on 2012-2013 WESTEST 2 mathematics outcomes to determine whether CCLC 
students (treatment) scored higher than students who did not receive the treatment 
(control). 

4. Eight tests (one per grade for Grades 4-11) examined the impact of group member- 
ship on 2013 WESTEST 2 reading/language arts outcomes to determine whether 
CCLC students (treatment) scored higher than students who did not receive the 
treatment (control). 
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RQ5 and RQ6 were addressed using a series of paired t tests. These analyses used 
time as the independent variable and students’ mean 2011-2012 and 2012-2013 WESTEST 
2 scale scores in mathematics and reading/language arts as the outcome variables. Each 
grade level and content area combination was tested independently. In sum, we conducted 
16 tests: 

5. Eight tests (one per grade for Grades 4-11) examined the impact of time on 
WESTEST 2 outcomes to determine whether CCLC students exhibited higher 
achievement in mathematics during the 2012-2013 school year when compared to 
their own mathematics results for the prior academic year. 

6. Eight tests (one per grade for Grades 4-11) examined the impact of time on 
WESTEST 2 outcomes to determine whether CCLC students exhibited higher 
achievement in reading/language arts during the 2012-2013 school year when 
compared to their own reading/language arts results for the prior academic year. 

RQ7 and RQ8 were addressed using a series of paired t tests. These analyses used 
time as the independent variable and mean 2011-2012 and 2012-2013 WESTEST 2 scale 
scores in mathematics and reading/language arts as the outcome variables. Each grade level 
and content area combination was tested independently. In sum, we conducted 16 tests: 

7. Eight tests (one per grade for Grades 4-11) examined the impact of time on 
WESTEST 2 outcomes to determine whether control group students exhibited higher 
achievement in mathematics during the 2012-2013 school year when compared to 
their own mathematics results for the prior academic year. 

8. Eight tests (one per grade for Grades 4-11) examined the impact of time on 
WESTEST 2 outcomes to determine whether control group students exhibited higher 
achievement in reading/language arts during the 2012-2013 school year when 
compared to their own reading/language arts results for the prior academic year. 

RQ9 and RQ10 were addressed using repeated measures analysis of variance (RM 
ANOVA) tests. These analyses used two predictor variables, group membership and time, as 
independent variables predicting the outcome of WESTEST 2 performance in mathematics 
and reading language arts. In these analyses we looked for a significant interaction effect to 
indicate one group scored differently from the other over time. In sum we conducted 16 
tests: 

9. Eight tests (one per grade for Grades 4-11) examined the interaction of group and 
time on WESTEST 2 mathematics outcomes to determine whether students in the 
treatment group scored significantly higher than students in the control group over 
time. 

10. Eight tests (one per grade for Grades 4-11) examined the interaction of group and 
time on WESTEST 2 reading/language arts outcomes to determine whether students 
in the treatment group scored significantly higher than students in the control group 
over time. 
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Results 

Results are presented below by research question. 

Statistics and Data Analysis 


RQl and RQ2 

Table l in Appendix A presents the results of independent t tests used to determine 
the statistical significance of differences in mathematics mean scale score gains between 
Group l (CCLC participants) and Group o (nonparticipants) for Grades 4-11. In no case 
were the observed differences statistically significant, leading us to reject H1-H8. See Figure 
1 for a graphical representation of mathematics gains by group and grade level. 



Figure 1. Difference in WESTEST 2 Mathematics Gain Scores by Group 
and Grade Level 


Table 2 in Appendix A presents the results of independent t tests used to determine 
the statistical significance of differences in reading/language arts mean scale score gains be- 
tween Group 1 (CCLC participants) and Group o (nonparticipants) for Grades 4-11. In no 
case were the observed differences statistically significant, leading us to reject H9-H16. See 
Figure 2 for a graphical representation of reading/language arts gains by group and grade 
level. 


6 I 21st Century Community Learning Centers 2013 


Results 



■ CCLC 

■ Control 


Figure 2. Difference in WESTEST 2 Reading/Language Arts Gain Scores by 
Group and Grade Level 


RQ3 and RQ4 

Table 3 in Appendix A presents the results of independent t tests used to determine 
the statistical significance of differences in 2012-2013 mathematics performance between 
Group 1 (CCLC participants) and Group o (nonparticipants) for Grades 4-11. In no case 
were the observed differences statistically significant, leading us to reject H17-H24. See Fig- 
ure 3 for a graphical representation of mathematics gains by group and grade level. 
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Figure 3. Difference in 2012-2013 Mathematics Performance by Group 
and Grade Level 
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Table 4 in Appendix A presents the results of independent samples t tests used to de- 
termine the statistical significance of differences in 2012-2013 reading/language arts per- 
formance between Group 1 (CCLC participants) and Group o (nonparticipants) for Grades 
4-11. In no case were the observed differences statistically significant, leading us to reject 
H25-H32. See Figure 4 for a graphical representation of reading/language arts gains by 
group and grade level. 



■ CCLC 

■ Control 


Figure 4. Difference in 2012-2013 Reading/Language Arts 
Performance by Group and Grade Level 


RQ5 and RQ6 

Table 5 in Appendix A presents the results of paired t tests used to determine the sta- 
tistical significance of differences in 2011-2012 and 2012-2013 mathematics performance 
for CCLC participants for Grades 4-11. The results were statistically significant for Grades 4- 
7, 9 and 11 leading us to accept H33-H36, H38, and H40. However, because results were not 
significant for Grades 8 and 10, H37 and H39 were rejected. Figure 5 provides a graphical 
representation of mathematics gains for the CCLC group. 



Figure 5. 2011-2012 to 2012-2013 Mathematics Gain Scores for 

CCLC Students by Grade 
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Table 6 in Appendix A presents the results of paired t tests used to determine the sta- 
tistical significance of differences in 2011-2012 and 2012-2013 reading/language arts per- 
formance for CCLC participants for Grades 4-11. The results were statistically significant for 
Grades 4-8 and 10 leading us to accept H41-H45 and H47. However, because results were 
not significant for Grades 9 and 11, we rejected H46 and H48. Figure 6 provides a graphical 
representation of reading/language arts gains for the CCLC group. 



Figure 6. 2011-2012 to 2012-2013 Reading/Language Arts Gain 

Scores for CCLC Students by Grade 

RQ7 and RQ8 

Table 7 in Appendix A presents the results of paired t tests used to determine the sta- 
tistical significance of differences in 2011-2012 and 2012-2013 mathematics performance 
for non-CCLC participants for Grades 4-11. The results were statistically significant for 
Grades 4-9 and 11 leading us to accept H49-H54 and H56. However, we rejected H55. Fig- 
ure 7 provides a graphical representation of mathematics gains for the CCLC group. 



Figure 7 . 2011-2012 to 2012-2013 Mathematics Gain Scores for 

non-CCLC Students by Grade 


21st Century Community Learning Centers 2013 / 9 


Results 


Table 8 in Appendix A presents the results paired t test analysis used to determine 
the statistical significance of differences in 2011-2012 and 2012-2013 reading/language arts 
performance for non-CCLC participants for Grades 4-11. The results were statistically signif- 
icant for Grades 4, 5, 6, and 7 leading us to accept H57-H60. However, we rejected H61- 
H64. Figure 8 provides a graphical representation of reading/language arts gains for the 
CCLC group. 



Figure 8. 2011-2012 to 2012-2013 Reading/Language Arts Gain Scores 

for non-CCLC Students by Grade 

RQ9 and RQ10 

Table 9 in Appendix A presents the within-subjects effect of time by grade for math- 
ematics analyses. In all grades, there was a statistically significant main effect for time indi- 
cating that, irrespective of group membership, all grades experienced statistically significant 
changes in mathematics performance from one year to the next. However, as indicated in 
Table 10 in Appendix A, none of the group*time interaction effects were statistically signifi- 
cant leading us to conclude that the two groups’ changes in mathematics performance did 
not differ significantly over time. Thus, we rejected H73-H80. Figure 9-Figure 16 in Ap- 
pendix B present group differences in mathematics achievement over time by grade level. 

Table 11 in Appendix A presents the within-subjects effect of time by grade for read- 
ing/language arts analyses. In all grades, there was a statistically significant main effect for 
time indicating that, irrespective of group membership, all grades experienced statistically 
significant changes in mathematics performance from one year to the next. However, as in- 
dicated in Table 12 in Appendix A, none of the group' :: time interaction effects were statisti- 
cally significant leading us to conclude that the two groups’ changes in reading/language arts 
performance did not differ significantly over time. Thus, we rejected H81-H88. Figure 17- 
Figure 24 in Appendix B present group differences in reading/language arts achievement 
over time by grade level. 
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Discussion 

Independent sample t tests sought to determine if CCLC students experience higher 
academic achievement gains in mathematics or reading/language arts than control group 
students from 2011-2012 to 2012-2013. Results showed no significant differences among 
these groups in either mathematics or reading/language arts scale score gains. With respect 
to mathematics, students in the CCLC program exhibited higher mean scale score gains in 
Grades 4, 8, 9, and 10; but the control group exhibited higher gains in Grades 5, 6, 7, and 11. 
However, in no case were these differences statistically or practically significant. With re- 
spect to reading/language arts, CCLC students exhibited higher mean scale score gains in 
Grades 5, 6, 8, 9 and 10; but control group students exhibited higher scale score gains in 
Grades 4, 7, 10 and 11. Thus, we rejected hypotheses H1-H32. 

The paired t-tests sought to determine if either group of students experienced signifi- 
cant academic achievement gains in mathematics or reading/language arts from 2011-2012 
to 2012-2013 when examined independently. CCLC students exhibited statistically signifi- 
cant mathematics gains in Grades 4, 5, 6, 7, 9 and 11; these students also exhibited statisti- 
cally significant gains in reading/language arts in Grades 4, 5, 6, 7, 8, and 10. These results 
were similar for control students who exhibited statistically significant mathematics gains in 
Grades 4, 5, 6, 7, 8, 9 and 11, and reading/language arts gains in Grades 4, 5, 6, and 7. Thus, 
we rejected hypotheses H61-H64. 

Repeated measures analyses of variance (ANOVA) were used to determine if student 
achievement in mathematics and reading/language arts varied by group over time. While we 
found significant main effects for time, there were no statistically significant interaction ef- 
fects among groups and time, indicating the groups did not experience differential gains 
over time. Thus, we rejected hypotheses H65-H96. However, it should be noted that the in- 
teraction effects did approach statistical significance in Grade 8 and Grade 10 reading/ 
language arts. In both cases, the CCLC group experienced stronger gains than the non-CCLC 
group. This finding may merit additional attention. 

Limitations 

The study had several important limitations that may limit our ability to draw defini- 
tive conclusions about the effectiveness of the CCLC program in producing academic 
achievement gains. First, the study was limited in that it only encompassed a single year of 
intervention. While we used baseline academic achievement to control for prior perfor- 
mance, it could be possible that additional time in CCLC is necessary to experience measur- 
able gains. Furthermore, and more problematic, our study was limited by the academic 
achievement data that were available, which included only Grades 4-11. There is currently 
no standardized academic achievement measure available for students K-2 and while Grade 
3 is a tested grade, prior test scores are not available to determine achievement gains. This is 
a critical limitation when one considers that 50% of students who participated in CCLC dur- 
ing the 2012-2013 school year were in Grades K-3. It should be noted that the West Virginia 
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Department of Education is considering developing assessments for K-2 in the future, which 
may allow for an evaluation study that addresses this deficiency. 

Recommendations 

To the extent possible, we will attempt to prepare next year’s edition of this report at 
the outset of the 2014-2015 school year, a time when the data are more actionable for CCLC 
program staff. Further, we will work with CCLC program staff to plan additional research to 
be conducted at the conclusion of the 2013-2014 school year to examine the impact of long- 
er-term participation in CCLC on student academic achievement outcomes. The study will 
examine outcomes for students who participated in CCLC for at least 2 academic years to 
determine if prolonged participation in the program produces statistically significant gains 
in achievement. 
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Appendix A 

This appendix includes detailed statistical information for all hypotheses tested. 


Table 1. H1-H8 Statistical Summary (Mathematics) 


Group 

Hypotheses 

Grade 

N 

Std. 

Mean deviation 

t 

df 

Sig. (2 
tail)* 

1-CCLC participants 

Hi 

4 

452 

9.46 

35.32 

.074 

902 

0.941 

O-Nonparticipants 


4 

452 

9.30 

30.43 




1-CCLC participants 

H2 

5 

423 

19.21 

37.92 

-.352 

844 

0.725 

O-Nonparticipants 


5 

423 

20.12 

37.20 




1-CCLC participants 

H3 

6 

242 

13.69 

33.76 

-1.277 

482 

0.202 

O-Nonparticipants 


6 

242 

17.93 

39.03 




1-CCLC participants 

H4 

7 

181 

11.85 

39.06 

-.233 

360 

0.816 

O-Nonparticipants 


7 

181 

12.77 

36.80 




1-CCLC participants 

H5 

8 

180 

-5.84 

40.01 

1.191 

358 

0.234 

O-Nonparticipants 


8 

180 

-10.81 

39.09 




1-CCLC participants 

He 

9 

67 

14.87 

43.58 

.055 

132 

0.956 

O-Nonparticipants 


9 

67 

14.45 

44.47 




1-CCLC participants 

H7 

10 

58 

-5.79 

55.14 

.595 

114 

0.553 

O-Nonparticipants 


10 

58 

-12.09 

58.66 




1-CCLC participants 

Hs 

11 

45 

18.00 

47.47 

-.859 

88 

0.393 

O-Nonparticipants 


11 

45 

27.51 

57.17 





*p<.05 for significance 


Table 2. H9-H16 Statistical Summary (Reading/Langauge Arts) 


Group 

Hypothesis 

Grade 

N 

Std. 

Mean deviation 

t 

df 

Sig. (2 
tail)* 

1-CCLC participants 

H9 

4 

452 

8.03 

28.65 

-.535 

902 

0.593 

O-Nonparticipants 


4 

452 

9.03 

27.66 




1-CCLC participants 

Hio 

5 

423 

8.30 

24.73 

1.199 

844 

0.231 

O-Nonparticipants 


5 

423 

6.14 

27.48 




1-CCLC participants 

Hu 

6 

242 

12.86 

31.47 

.586 

482 

0.558 

O-Nonparticipants 


6 

242 

11.31 

26.67 




1-CCLC participants 

Hl2 

7 

181 

5.64 

28.74 

-.683 

360 

0.495 

O-Nonparticipants 


7 

181 

8.03 

37.50 




1-CCLC participants 

Hl3 

8 

180 

6.22 

25.14 

1.818 

358 

0.070 

O-Nonparticipants 


8 

180 

1.46 

24.56 




1-CCLC participants 

Hl4 

9 

67 

6.40 

29.41 

.576 

132 

0.566 

O-Nonparticipants 


9 

67 

3.28 

33.18 




1-CCLC participants 

Hl5 

10 

58 

15.78 

37.07 

1.82 

114 

0.071 

O-Nonparticipants 


10 

58 

4.34 

30.24 




1-CCLC participants 

Hl6 

11 

45 

-12.07 

65.06 

-.548 

88 

0.585 

O-Nonparticipants 


11 

45 

-5.33 

50.59 




*p<.05 for significance 
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Table 3. H17-H24 Statistical Summary (Mathematics) 

Std. Sig. (2 


Group 

Hypothesis 

Grade 

N 

Mean 

deviation 

t 

df 

tail)* 

1-CCLC participants 

Hl7 

4 

452 

583.20 

47.30 

-.903 

902 

0.367 

O-Nonparticipants 


4 

452 

585.93 

43.45 




1-CCLC participants 

Hl8 

5 

423 

601.40 

51.08 

.194 

844 

0.847 

O-Nonparticipants 


5 

423 

600.72 

50.87 




1-CCLC participants 

Hl9 

6 

242 

616.85 

49.91 

-.394 

482 

0.694 

O-Nonparticipants 


6 

242 

618.53 

43.85 




1-CCLC participants 

H20 

7 

181 

622.45 

51.12 

-.134 

360 

0.893 

O-Nonparticipants 


7 

181 

623.15 

49.27 




1-CCLC participants 

H21 

8 

180 

634.66 

55.55 

-.068 

358 

0.946 

O-Nonparticipants 


8 

180 

635.08 

61.29 




1-CCLC participants 

H22 

9 

67 

650.87 

53.36 

.003 

132 

0.997 

O-Nonparticipants 


9 

67 

650.84 

52.75 




1-CCLC participants 

H23 

10 

58 

646.72 

56.07 

.612 

114 

0.446 

O-Nonparticipants 


10 

58 

640.26 

57.80 




1-CCLC participants 

H24 

11 

45 

653.87 

40.96 

-.561 

88 

0.576 

O-Nonparticipants 


11 

45 

659.02 

46.01 





*p<.05 for significance 


Table 4. H25-H32 Statistical Summary (Reading/Language Arts) 

Std. Sig. (2 


Group 

Hypothesis 

Grade 

N 

Mean 

deviation 

t 

df 

tail)* 

1-CCLC participants 

H25 

4 

452 

442.55 

41.78 

-1.932 

902 

0.054 

O-Nonparticipants 


4 

452 

447.86 

40.95 




1-CCLC participants 

H26 

5 

423 

450.36 

38.03 

.807 

844 

0.420 

O-Nonparticipants 


5 

423 

448.17 

41.14 




1-CCLC participants 

H27 

6 

242 

467.19 

45.49 

.296 

482 

0.767 

O-Nonparticipants 


6 

242 

466.07 

36.90 




1-CCLC participants 

H28 

7 

181 

467.50 

36.68 

-.214 

360 

0.830 

O-Nonparticipants 


7 

181 

468.38 

41.66 




1-CCLC participants 

H29 

8 

180 

480.23 

38.29 

.822 

358 

0.411 

O-Nonparticipants 


8 

180 

476.78 

41.25 




1-CCLC participants 

H30 

9 

67 

483.25 

51.48 

.576 

132 

0.565 

O-Nonparticipants 


9 

67 

477.39 

65.47 




1-CCLC participants 

H31 

10 

58 

495.62 

51.02 

.765 

114 

0.446 

O-Nonparticipants 


10 

58 

488.43 

50.21 




1-CCLC participants 

H32 

11 

45 

460.02 

79.72 

-956 

88 

0.342 

O-Nonparticipants 


11 

45 

475.78 

76.65 





*p<.05 for significance 
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Table 5. H33-H40 Statistical Summary 

CCLC participants 

Hypothesis 

Grade 

N 

Mean diff 

Std. 

deviation 

t 

df 

Sig. (2 
tail)* 

Mathematics 

H33 

4 

452 

9.458 

35.31 

5.694 

451 

.000* 

Mathematics 

H34 

5 

423 

19.21 

37.96 

10.421 

422 

.000* 

Mathematics 

H35 

6 

242 

13.690 

33.76 

6.307 

241 

.000* 

Mathematics 

H36 

7 

181 

11.845 

39.06 

4.080 

180 

.000* 

Mathematics 

H37 

8 

180 

-5.844 

40.01 

-1.960 

179 

.052 

Mathematics 

H38 

9 

67 

14.866 

43.58 

2.792 

66 

.007* 

Mathematics 

H39 

10 

58 

-5.793 

55.14 

-.800 

57 

.427 

Mathematics 

H40 

11 

45 

18.000 

47.466 

2.54 

44 

.015* 

*p<.05 for significance 


Table 6. H41-H48 Statistical Summary 


CCLC participants Hypothesis 

Grade 

Std. 

N Mean diff deviation 

t 

df 

Sig. (2 
tail)* 

Reading/language arts 

H41 

4 

452 

8.029 

28.65 

5.957 

451 

.000* 

Reading/language arts 

H42 

5 

423 

8.300 

24.73 

6.903 

422 

.000* 

Reading/language arts 

H43 

6 

242 

12.860 

31.47 

6.357 

241 

.000* 

Reading/language arts 

H44 

7 

181 

5.635 

28.74 

2.638 

180 

.009* 

Reading/language arts 

H45 

8 

180 

6.217 

25.14 

3.318 

179 

.001* 

Reading/language arts 

H46 

9 

67 

6.403 

29.41 

1.782 

66 

.079 

Reading/language arts 

H47 

10 

58 

15.776 

37.07 

3.241 

57 

.002* 

Reading/language arts 

H48 

11 

45 

-12.067 

65.06 

-1.244 

44 

.220 

*p<.05 for significance 


Table 7. H49-H56 Statistical Summary 


Non-CLC Participants 

Hypothesis 

Grade 

Std. 

N Mean diff deviation 

t 

df 

Sig. (2 
tail)* 

Mathematics 

H49 

4 

452 

9.296 

30.43 

6.49 

451 

.000* 

Mathematics 

H50 

5 

423 

20.121 

37.20 

11.12 

422 

.000* 

Mathematics 

H51 

6 

242 

17.926 

39.03 

7.144 

241 

.000* 

Mathematics 

H52 

7 

181 

12.773 

36.80 

4.67 

180 

.000* 

Mathematics 

H53 

8 

180 

-10.811 

39.09 

-3.710 

179 

.000* 

Mathematics 

H54 

9 

67 

14.448 

44.47 

2.659 

66 

.010* 

Mathematics 

H55 

10 

58 

-12.086 

58.66 

-1.569 

57 

.122 

Mathematics 

H56 

11 

45 

27.511 

57.17 

3.23 

44 

.002* 


*p<.05 for significance 
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Table 8. H57-H64 Statistical Summary 


Non-CCLC participants 

Hypothesis 

Grade 

N 

Mean 

diff 

Std. 

deviation 

t 

df 

Sig. (2 
tail)* 

Reading/language arts 

H57 

4 

452 

9.031 

27.66 

6.94 

451 

.000* 

Reading/language arts 

H58 

5 

423 

6.144 

27.48 

4.60 

422 

.000* 

Reading/language arts 

H59 

6 

242 

11.306 

26.67 

6.59 

241 

.000* 

Reading/language arts 

H60 

7 

181 

8.033 

37.50 

2.88 

180 

.004* 

Reading/language arts 

H61 

8 

180 

1.456 

24.56 

.795 

179 

.428 

Reading/language arts 

H62 

9 

67 

3.284 

33.18 

.810 

66 

.421 

Reading/language arts 

H63 

10 

58 

4.345 

30.24 

1.094 

57 

.278 

Reading/language arts 

H64 

11 

45 

-5.33 

50.59 

-.707 

44 

.483 


*p<.05 for significance 


Table 9. Within Subjects Effects for Time (Mathematics Analyses) 

Type III sum 


Grade 

Source 

of squares 

df 

Mean square 

F 

Sig.* 

4th grade 

H65 

Time Huynh-Feldt 

39745.315 

1.000 

39745.315 

73.166 

.000* 

5th grade 

Hee 

Time Huynh-Feldt 

163587.334 

1.000 

163587.334 

231.931 

.000* 

6th grade 

H67 

Time Huynh-Feldt 

60472.935 

1.000 

60472.935 

90.811 

.000* 

7th grade 

H68 

Time Huynh-Feldt 

27425.326 

1.000 

27425.326 

38.091 

.000* 

8th grade 

H69 

Time Huynh-Feldt 

12483.339 

1.000 

12483.339 

15.957 

.000* 

9th grade 

H 70 

Time Huynh-Feldt 

14392.896 

1.000 

14392.896 

14.850 

.000* 

10th grade 

H 71 

Time Huynh-Feldt 

4635.211 

1.000 

4635.211 

2.861 

.094 

11th grade 

H 72 

Time Huynh-Feldt 

23301.689 

1.000 

23301.689 

16.881 

.000* 


*p<.05 for significance 
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Table 10. Interaction Effects for Time*Group (Mathematics Analyses) 

Type III sum 


Grade 

Source 

of squares 

df 

Mean square 

F 

Sig.* 

4th grade 

H73 

Time*group 

3565.554 

1.000 

3565.554 

1.239 

.266 

5th grade 

H74 

Time*group 

543.547 

1.000 

543.547 

.130 

.718 

6th grade 

H75 

Time*group 

45.993 

1.000 

45.993 

.012 

.912 

7th grade 

H76 

Time*group 

10.696 

1.000 

10.696 

.003 

.959 

8th grade 

H77 

Time*group 

1513.800 

1.000 

1513.800 

.302 

.583 

9th grade 

H78 

Time*group 

2.149 

1.000 

2.149 

.000 

.984 

10th grade 

H79 

Time*group 

638.901 

1.000 

638.901 

.154 

.696 

11th grade 

H80 

Time*group 

7.200 

1.000 

7.200 

.002 

.961 

*p<.05 for significance 


Table 11. 

Within Subjects Effects for Time (Reading/Language Arts Analyses) 



Grade 

Source 


Type III sum 
of squares 

df 

Mean 

square 

F 

Sig.* 

4th grade 

Time 

Huynh-Feldt 

32886.903 

1.000 

32886.903 

82.937 

.000* 

H81 

Time*group 

Huynh-Feldt 

113.501 

1.000 

113.501 

.286 

.593 

5th grade 

Time 

Huynh-Feldt 

22063.889 

1.000 

22063.889 

64.568 

.000* 

H82 

Time*group 

Huynh-Feldt 

491.574 

1.000 

491.574 

1.439 

.231 

6th grade 

Time 

Huynh-Feldt 

35329.653 

1.000 

35329.653 

83.035 

.000* 

H83 

Time*group 

Huynh-Feldt 

146.050 

1.000 

146.050 

.343 

.558 

7th grade 

Time 

Huynh-Feldt 

8453.972 

1.000 

8453.972 

15.151 

.000* 

H84 

Time*group 

Huynh-Feldt 

260.160 

1.000 

260.160 

.466 

.495 

8th grade 

Time 

Huynh-Feldt 

2648.835 

1.000 

2648.835 

8.581 

.004* 

H85 

Time*group 

Huynh-Feldt 

1020.068 

1.000 

1020.068 

3.305 

.070 

9th grade 

Time 

Huynh-Feldt 

1571.646 

1.000 

1571.646 

3.198 

.076 

H86 

Time*group 

Huynh-Feldt 

162.989 

1.000 

162.989 

.332 

.566 

10th grade 

Time 

Huynh-Feldt 

5870.211 

1.000 

5870.211 

10.260 

.002* 

H87 

Time*group 

Huynh-Feldt 

1894.694 

1.000 

1894.694 

3.311 

.071 

11th grade 

Time 

Huynh-Feldt 

3406.050 

1.000 

3406.050 

2.006 

.160 

H88 

Time*group 

Huynh-Feldt 

510.050 

1.000 

510.050 

.300 

.585 


*p<.05 for significance 
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Table 12. Interaction Effects for Time*Group (Reading/Language Arts Analyses) 

Type III sum 


Grade 

Source 

of squares 

df 

Mean square 

F 

Sig.* 

4th grade 

H89 

Time*group 

10480.425 

1.000 

10480.425 

3.640 

.057 

5th grade 

H90 

Time*group 

531.149 

1.000 

531.149 

.202 

.653 

6th grade 

H91 

Time*group 

27.785 

1.000 

27.785 

.010 

.921 

7th grade 

H92 

Time*group 

17.950 

1.000 

17.950 

.006 

.938 

8th grade 

H93 

Time*group 

205.868 

1.000 

205.868 

.076 

.783 

9th grade 

H94 

Time*group 

1242.272 

1.000 

1242.272 

.233 

.630 

10th grade 
H95 

Time*group 

126.039 

1.000 

126.039 

.032 

.858 

11th grade 

Time*group 

6906.806 

1.000 

6906.806 

.900 

.345 


H96 

*p<.05 for significance 
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This appendix presents graphical representations of the differences observed in stu- 
dent achievement over time by group. 


Mathematics 



CCLC 

Control 


Figure 9. Fourth Grade Year-to-Year Mathematics Achievement by Group 



CCLC 

Control 


Figure 10. Fifth Grade Year-to-Year Mathematics Achievement by Group 
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Figure 16. Eleventh Grade Year-to-Year Mathematics Achievement by Group 
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Reading/Language Arts 
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•Control 
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Figure 1 7. Fourth Grade Year-to-Year Reading/Language Arts Achievement by Group 
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Figure 18. Fifth Grade Year-to-Year Reading/Language Arts Achievement by Group 
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Figure 19. Sixth Grade Year-to-Year Reading/Language Arts Achievement by Group 
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Figure 20. Seventh Grade Year-to-Year Reading/Language Arts Achievement by Group 
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Figure 21. Eighth Grade Year-to-Year Reading/Language Arts Achievement by Group 
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